Skip to content

[CI] Skip MegaMoE v2 multi-GPU tests - #5003

Merged
gyohuangxin merged 1 commit into
mainfrom
xihuang/skip-megamoe-multigpu-ci
Aug 26, 2026
Merged

[CI] Skip MegaMoE v2 multi-GPU tests#5003
gyohuangxin merged 1 commit into
mainfrom
xihuang/skip-megamoe-multigpu-ci

Conversation

@gyohuangxin

@gyohuangxin gyohuangxin commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary:

  • skip bench_mega_moe_v2.py and test_mega_moe_v2.py in the multi-GPU CI sweep
  • remove the MegaMoE/Mori-specific multi-GPU install dependencies added for those tests
  • delete the now-unreachable special command branches for the skipped tests

Root cause:

  • recent main runs time out in Multi-GPU Tests after test_mega_moe_v2.py starts torchrun and stops making progress
  • bench_mega_moe_v2.py also fails early because Mori JIT cannot find infiniband/verbs.h
  • keeping both tests in the required 8-GPU sweep blocks main recovery and holds the scarce 8-GPU runner until timeout

Validation:

  • git diff --check
  • bash -n .github/scripts/aiter_test.sh

@gyohuangxin
gyohuangxin requested review from a team and a lite review from Copilot August 26, 2026 03:02
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5003 --add-label <label>

PR title tags:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf] and op tags like [MLA] are left untouched. Add the no-auto-title label to opt this PR out of title tagging.

Copilot AI reviewed Aug 26, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the main-branch 8‑GPU CI sweep to skip the MegaMoE v2 multi-GPU tests that are currently causing timeouts/hard failures, and removes the now-unneeded Mori/MegaMoE-specific install and command branching that supported those tests.

Changes:

  • Skip bench_mega_moe_v2.py and test_mega_moe_v2.py in the multi-GPU test runner script.
  • Remove the Mori/MegaMoE-specific dependency installation from the multi-GPU workflow setup.
  • Delete special-case torchrun command branches that are now unreachable due to the skip list.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/aiter-test.yaml Removes Mori/MegaMoE-specific install steps from the multi-GPU CI environment setup.
.github/scripts/aiter_test.sh Adds MegaMoE v2 tests to the multi-GPU skip list and removes their special-case execution branches.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/aiter-test.yaml
Comment on lines 1007 to 1011
pip install -r requirements.txt
pip install --upgrade pandas pyzmq einops numpy==1.26.2
pip install --upgrade 'pybind11>=3.0.1'
pip install --upgrade 'ninja>=1.11.1'
pip install tabulate
@gyohuangxin
gyohuangxin merged commit f05d69d into main Aug 26, 2026
27 checks passed
@gyohuangxin
gyohuangxin deleted the xihuang/skip-megamoe-multigpu-ci branch August 26, 2026 03:28
@IzacharyI IzacharyI mentioned this pull request Oct 23, 2026
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gyohuangxin